The Crowd vs. the Lab: A Comparison of Crowd-Sourced and University Laboratory Participant Behavior

There are considerable differences in remuneration and environment between crowd-sourced workers and the traditional laboratory study participant. If crowd-sourced participants are to be used for information retrieval user studies, we need to know if and to what extent their behavior on information retrieval tasks differs from the accepted standard of laboratory participants. With both crowd-sourced and laboratory participants, we conducted an experiment to measure their relevance judging behavior. We found that while only 30% of the crowd-sourced workers qualified for inclusion in the final group of participants, 100% of the laboratory participants qualified. Both groups have similar true positive rates, but the crowd-sourced participants had a significantly higher false positive rate and judged documents nearly twice as fast as the laboratory participants.